SerializerManager

Caution

When SparkEnv is created (either for the driver or executors), it instantiates SerializerManager that is then used to create a BlockManager.

It automatically selects the best serializer which is KryoSerializer whenever possible or falls backs to the input defaultSerializer serializer.

The common idiom in Spark’s code is to access the current SerializerManager using SparkEnv.

SparkEnv.get.serializerManager

Caution

Caution

Caution

Table 1. Settings
Name	Default value	Description
`spark.io.encryption.enabled`	`false`	The flag to enable IO encryption
`spark.shuffle.compress`	`true`	The flag to control whether to compress shuffle output when stored
`spark.rdd.compress`	`false`	The flag to control whether to compress RDD partitions when stored serialized.
`spark.shuffle.spill.compress`	`true`	The flag to control whether to compress shuffle output temporarily spilled to disk.
`spark.block.failures.beforeLocationRefresh`	`5`

results matching ""